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ABSTRACT 



A Comparison of Two Approaches in Factor Studies o f_ 
Students Ratings of Courses and Instructors 

Donald W. Carver, Measurement and Evaluation Center, 
The University of Texas at Austin 
Paul G. Liberty, Jr., Measurement and Evaluat ion 

The University of Texas ^aLAustin-----'"'*"''*'^ ^ 

Results of two factor analyses of 3772 student ratings of engineering courses and instructors, 
obtained by using alternative step facti^ring criteria, were compared on the basis of six criteria. 
Results indicated that the ten factor solution obtained by using a step criterion of accounting for 
70 - 75% of the total variance as opposed to a five factor solution containing only factors having 
an eigenvalue greater than 1.0 accounted for a higher percentage of the total variance, clarified 
factor structure, provided more useful information for administration decision-making, and 
supported the view that a 1.0 eigenvalue step criterion is unnecessarily restrictive in certain 
situations. 



INTRODUCTION 



Previous studies of student ratings of courses and instructors 
have established the value of applying factor analysis to obtain a 
reduced set of evaluation dimensions (e.g., Eendig, 1954; Spencer and 
Aleamoni, 1969; Brooks and Tarver, 1971). The emphasis in these 
studies has been on identifying a minimal set of factors which repre- 
sent a general set of dimensions in student^ratings. But when considered 
from the viewpoint of the evaluative needs of instructors and 
administrators in specific academic areas this generalized approach 
may not be adequate. 

Previous factor analytic studies of student ratings have 
normally yielded small sets of factors based on a stop factoring 
criterion having eigenvalues greater than 1.0 (Kaiser, 1960; Harman, 
1960). Other criterion have been suggested such as Cattell's (1966) 
"scree test" and Horn's (1965) method which uses Guttman's latent- 
root-one lower bound estimate as a psychometric upper bound. Horst 
(1965, 1968) has suggested the alternative of determining the number 
of factors based on the percentage of total variance extracted where 
this percentage is determined with respect to a criteria such as 
instrument reliability. 

The rationale here is that true variance is being factored. 
This latter alternative provides a method for obtaining a larger set 
of factors for explanatory purposes. For evaluation needs in specific 
academic areas more often the greater sin is to take out too few 
factors than too many. Too few factors has the effect of distorting 



the factor structure — so that in terms of interpreting factors the 
greater error occurs in the too-few situation. Taking .70 to .75 as 
a reasonable estimate of the reliability of student rating forms this 
study posed the question of whether the stop-factoring criterion of 
75% of the total variance provided a set of factors with a greater 
explanatory power than a set obtained, by a stop criteria of an 
eigenvalue of one. 

The purpose of this study was to determine: 1) In comparing 
the two approaches, -rhrr^^^su^. i n^y Qfjjjp total va\-iance stop criterion 
yield additional factors which are mathematicall5'"-44;^tinct and 
psychologically interpretable? 2) Will additional f actors^-^J^rify 
ambiguous items, identify clusters with excessively redundant it^Jtvs 
and indicate small clusters needing additional items? 3) Is a larger 
set of distinctly interpretable factors more useful for evaluation 
of courses and instructors? 4) Does the higher stop-criterion provide 
more useful information for form revisions? 

DATA SOURCES AND ANALYSIS METHODS 

The student ratings of courses and instructors used as data 
for these analyses were obtained on the University of Texas at Austin 
College of Engineering Course-Instructor Survey Form (Appendix A) . 
Thirty-five (items 5-39) of a total of forty-six items were included. 
The remaining items were excluded because they contained either 
categorical information or less relevant content. Each item was 
scored on a five-point rating scale which was coded: 1 - excellent, 
2 - above average, 3 - average, 4 - below average, and 5 - unsatisfactory. 



Ratings were obtained from 3,772 students in approximately 
200 College of Engineering courses during the fall semester of 1971, 
Graduate and undergradCiate courses taught by all levels of instructors 
were included although the classes included only those instructors 
who voluntarily agreed to participate in the survey. The survey forms 
were administered by students during one of the last regularly 

scheduled class meetings. |The instructor was asked to leave the 

1 

room while the students ananymously completed the survey form. 

Using individual responses as a data base (as opposed to 
class means)/ the ratings were intercorrelated and subjected to two 
principal components, unit diagonals, factor analyses. A varimax 
rotation was then performed on the principal components factors 
(Kaiser, 1959) . These analyses were performed on the 6600 computer 
system at the UT Austin Computation Center using program FACTOR 
(Veldman, 1967) . A missing data option was utilized which allowed 
nfediiS, standard deviations and correlation coefficients between pairs 
of items to be based only on the ratings of students who responded 
to those items or pairs of items. The first analysis was performed 
using standard stop-factoring criterion of obtaining only those 
factors whose eigenvalues are greater than one. The second analysis 
used a stop-criterion of 75% of the total variance. The first 
analysis yielded five factors and the second analysis resulted in 
ten factors. 

RESULTS 

The results obtained from using a stop-criterion of 70-75% 
of the total variance as opposed to an eigenvalue of 1.0 can be com- 
pared for 1) percent of total variance accounted for, 2) mathematical 
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Stability, 3) factor clarity and distinctiveness, 4) redundancy (ex- 
cessive items in a factor, 5) meaningful interpretability and 6) utility 
of results for administrative decision-making. The factor solutions can 
also be examined to determine implications for form revision. 

As indicated in Table 1 the percent of trace or total vari- 
ance accounted for by the proportion of variance stop-criterion is 
74.88% while the eigenvalue 1.0 stop-criterion yielded factors accounting 
for 64.55% of the total variance. Thus a difference of 10% more vari- 
ance is accounted for th the larger set of factors. 

Table 2 describes the relationship of the five factor set 
to the new factors emerging in the ten factor set. For example factor 5 
in the five factor set splits into factor* 5,7 and 8"in the ten tactor 
set. These relationships are not exact in that single items from other 
factors may be involved as will be seen later (Table 6) but Table 1 
does serve the purpose of showing how the interpretability of factors is 
is increased by the larger set of factors. For example it would be 
difficult to interpret the combination of fc^ctors 5,7 and 8 which emerged 
from the original factor 5 by one interpretive name* It would have to com- 
bine effect on student, course content and course assignments. A close 
examination of the individual items listed in Table 3 for each factor 
of the ten factor set will reveal that each new factor clarifies the 
interpretability of the factor structure. 

As may be seen in Table 2 two factors (3 and 4) remained stable 
while the other three factors (1,2 and 5) yielded five additional factors. 
All five of the new factors were meaningfully interpretable. 

Tables 4 and 5 show the primary factor loadings (highest load- 
ing of an item) on the rotated factor solutions of the five and ten fac- 
tor solutions respectively. If the mathematical distinctiveness of a 
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Factor 1 


i. Inscructor openness 

6. Instructor interest in students 


Factor 2 
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2. Instructor organization .s. clarity * 

i 

9. Instructor speaking & writing ; 
10. Course application (one item) | 


Factor 3 


■ ' i 

3. Course grading 


Factor 4 


4. Instructor enthusiasm 


Factor 5 


5. Effects on students 

7. Course content 

8, Course assignments 



TABLE 3 

FACTOR- ITEM GROUPINGS 
TEN FACTOR SOLUTION 

Factor I; Instructor Openness 

18. He encourages class discussion, 

21. He invites criticism of his own ideas. 

35. He is sensitive to the responses of the class, encourages 

student participation, and welcomes questions and discussion. 

Factor II ; Instructor Organization & Clarity 

14. He presents the subject matter in an organized, easily understood 
fashion. 

34. He makes himself clear, states objectives, summarizes major 

points, presents material in an organized manner and provides 
emphasis. 

j 

12. The instructor explains the material clearly. 

13. He identifies what he considers to be the important concepts in 
the course. 

11. The instructor makes clear to the students what the educational 
objectives of the course are. 

22. He usually knows whether or not the class is understanding him. 

33. The instructor has command of the subject, presents material 

in an analytic way, contrasts points of view, discusses current 
developments, and relates topics to other areas of knowledge. 

^26. He answers student questions effectively in class. 

23. He keeps well informed about the progress of the class. 
Factor III ; Course Grading 

39. Homework is usually graded and returned promptly. 
38. The grader for this course is accessible and helpful. 



Factor IV : Instructor Enthusiasm 

30. He seems to have self-confidence, 

29. He seems to enjoy teaching this course. 

28. He is a dynamic and energetic person. 

37. He enjoys teaching, is enthusiastic about his subject, makes 
the course exciting, and has self-confidence. 

16. He inspires confidence in his knowledge of the field of this 
course. 

Factor V ; Effect on Students 

32. I have developed an increased appreciation for this subject 
after taking this course. 

8. In comparison with other Engineering coul:^s I have taken at 
UT Austin, this course has been: \ 



19. He has motivated me to do my best possible work in this cdxji^e. 
Factor VI : Instructor Interest in Students 

25. He is available to students outside of class. 

36. He is available to and friendly toward students, is interested 

in students as individuals, is himself respected as a person and 
is valued for advice not directly related to the course. 

24. He seems to have a genuine interest in students. 

27. He appears to be fair and impartial in his contacts with students • 
Factor VII : Course Content 

6. The textbook in this course is a good one. 

5. The content of this course is professionally up-co-date. 
Factor VIII : Course Assignments 

17. His assignments and his expectations on homework are clear. 

20. He has obtained sufficient evidence to evaluate accurately 
my achievement in this course. 

7. The homework assignments support and enhance the learning 
experience in this course. 



31. He has enhanced my skills in thinking and in pro^ 




solving. 



Factor IX ; Instructor Speaking & Writing 

10. He writes on the chalkboard legibly and large enough to be 
seen clearly. 

9. The instructor speaks clearly and can be easily understood. 
Factor X : Course Application 

15. He relates course material to engineering applications by 
giving appropriate examples and illustrations. 
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TABLE 4 

PRIMARY* FACTOR . SnT . UTKW ON 
ROTATED FIVE FACTOR SOLUTION 
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*Primary loading = highest loading of an item 

**non-distinctive loading = item having a non-primary loading 

greater then lowest primary loading for 
that factor. 

= primary loading of an item having a non-distinctive loading on 
another item. 
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factor is defined as the absence in a factor of any item with a non- 
primary loading greater than the lowest primary loading for that factor, 
then it can be seen in Table 5 that four of the five new factors are 
mathematically distinct. Although the non-distinct factor (10) is a one 
item split-off from factor 2 (where it continues to have a non-distinct 
loading) , it is meaningfully interpretable from the original factor. 

An item which has a non-distinctive loading (i.e. a non- 
primary loading greater than the lowest primary loading for that fac- 
tor) is an ambiguous item. In Table 4 it is seen that the five 
factor solution contains two ambiguous items - item 26 and item 33. In 
Table 5 it can be seen that the ambiguity of these items is eliminated 
although one new ambiguity emerges (item 15 which becomes factor 10) . 
Overall the ambiguity of items appears to be reduced by the larger set 
though not conclusively. 

Table 6 diagrams the movement of primary loadings of items 
between the two factor sets. The stability of factors 3 and 4 is clearly 
seen in that factor 3 remains identically the same and factor 4 loses 
only one item. The clarification of factors 1 and 5 is also clearly seen 
Ln the forming of three distinctively new factors. Just in terms of item 
movement factor 2 does not appear to be improved although, as previously 
indicated; the new cluster of items is nore easily interpreted. Concern- 
ing item redundancy it can be noted in Table 6 that only one factor (2) 
in the ten factor set remained with greater than 7 items while three 
factors (1;2 & 5) originally contained greater than 7 items in the five 
factor set. With the exception then of one factor, redundancy was 
greatly improved. Table 6 also shows the need for additional items in 
factors 3,7, 9 and 10 of the larger set. 



CONCLUSIONS AND SIGNIFICANCE 



From the results of this study the following conclusions can 
be made: 1) The stop-criterion of 75% of the total variance yielded 
factors covering a higher percentage of the total variance. The distinc- 
tiveness and interpretability of the factors added considerably to the 
interpretive clarity of the survey, 2) The clarification of factors 
provided by the larger set of factors proved more useful for evaluation 
purposes than a smaller more general set of factors. 3) Useful infor- 
mation was obtained for revising the fomn both with respect to ambiguous 
and redundant items and to distinctive clusters needing additional items. 
4) Follow-up studies with data bases in the same college curriculum area 
and in other curriculum areas are needed to determine the stability of 
these findings, to determine the nature of factor results when all 
faculty members participate in the survey and to determine the com- 
parability of results when class averages rather than individual stu- 
dent responses are factored. 

The significance of the study lies in providing supporting 
evidence for the value of considering the use of a factoring procedure 
based on a 70-7 5% stop-criterion for course-instructor survey studies. 
From the viewpoint of providing information to administrators and in- 
structors, the use of a stop-criterion of eigenvalue 1.0 is seen as 
unnecessarily restrictive. Course- instructor survey studies may more 
appropriately be geared toward a factor methodology that provides more 
situation-specific factors. The study calls attention to the difference 
in results that can be obtained by the use of two different factor 
extraction criteria. Work in the field has seen "blind" adoption of 
the eigenvalue 1.0 criterion with little consdieration for the purpose 



of factor studies. From the viewpoint of determining general laws of 
behavior, the eigenvalue 1.0 criterion with individual student responses 
seems appropriate. Conversely, the "applications" viewpoint argues for 
the more "liberalized" cirterion based on the estimated reliability of 
the rating instrument. This approach yields more specific factors that 
are valuable to administrators and instructors as they make decisions 
about the content of courses and the quality of instruction. From the 
theoretical point of view as well, the factor methodology has been 
biased toward eigenvalue 1.0 interpretations. Thus, this study is seen 
as having, possibly, important theoretical ramifications as well. 
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D = Below aveiage 
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' PJ!:!?:§iL!ry^PJAp 

5 The contcn: ot Miis '*ou:se is r*otes^:onaHy up-to-date 

6 The textbook *n this 4,ou'se ;s a cood v;ne 

7 The hontev^ork ass«^nn\ents suppou ana enhance th- leannng experience in ^his couise 

H In conipanson ^Mih o^he- Engmetnng couises I have taken at UT Austin, this couise has been: 
^ -^OiL'rJQR J: 

> The uK<tHicto' speaks clearly anC can oe ea^Iy un.'Jerstood 
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25 He js avai'aole lo .<JuOcn'^ ou -uie of class 

26. He ansvscfs >tuJoin ^juesnoib cilccnvely ui class 

11 Ho appea:s lo be fair and nrK,*a'i.al in rus contatis wjih students 

28 He i.s d .lynaniic jocI cncpjetu jmson 

20. He secMiis erijoy '.cachuiL* iSi.s coiir.-iC 

30 He sccm to l.ave s-l! confianco 

31 He lias titliancea my ikilis in tl/nkinj, an.! :n pjoblcni solving 

32 f have dovdopcu an iji::ea?ec' af ^/f/d'a^'on io lh»^ suoievi afjcc takm^ ih;^ coiuj. 

OVtR.AU aLI' AJ10N Oi _ IN^TRL^T^^^ 

33 The liibtuiLiu; has coninianJ oi .lie .subiCct. pic^cni:* niatcsal in an analytic way. ^oni<a^is point.\ ol vicvv. OiSLiisses cuiicnt 
dcvelopmenis and lelates topics to othei areas of knowiedt:e. 

34. He makes hinisdi cleai, states «jf/icctives, suniman^es niajoj pointb, piesent niateiial in an o/i-ani/td manner and pioMdci» 
emphasis 

35 He IS sensinu' to ihe responses of the class, encourages student paiticipation, and vvclcontes tjuestions and discussion 

36 He IS available to and friendly toward studenis, is inteiested m studenti* as individuals, is niinself lospected as a per.Mm and 
valued foj advice not directly ;claJed to Hie course 

37 He ciijoys leachmu, 's enihusiastJc about his subicct. makes the course exciting, and has sclf-cunfidence. 
GRAPER tVALlJATfON 

38 The grader for thio C(niise is accessible and helpful 

39 Homcwoik is usually naded and icrunied promptly. 
ADDITIONAL ITHiMS 

For Items 4046, clioose the appropriate response from those given for die item 
40. The number of qinzyes given during the course is: 

A = Too huge B = About right C = Too small 

41 The qaiz7.es are gcncially* 

A = Too difficult B = About iigiit in difficulty C = Too easy 

42 The quizzes aic generally: 

A = Too long B ~ About iight in length C = Too short 

43 The pre-iequisities Ibi this course are. 

A = Not sufficient: ihere should be mote D = More than are needed; theie should be fevvei 

B = Veiy appropriate F = Non-existent; there are none, and noiie are needed 

C = Adequate 

44 For each iiour of class, the average amount of time I spend on this course outside of class iS about: 
A = 4 hours 01 more D = 1 hour 

B = 3 houis E, = Less than 1 hour 

C = 2 hours 

45 The amouiu of contend of th^s course is: 

A - Too extensive to covei in one semestei 

B = About light foj one semester 

C = Not extensive enougli to require a whole semester 

46. The classroom in which this course meets is: 

A - Good 
B = Adequate 

C = Needs improvement (In the COMMENTS section of the answer sheet, please state how it should be improved.) 
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